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ABSTRACT 



Research related to the implementation and impact of the 
efficacy process on teachers and students in four Sacramento (California) 

City Unified School District pilot schools over a 4-year period is described. 
The study investigated three research evaluation questions: (1) the extent of 

teachers' implementation of the efficacy process; (2) the degree of 
participating students' growth in personal efficacy; and (3) the degree to 
which participating schools showed improved performance on the Comprehensive 
Tests of Basic Skills (CTBS-U) . Students enrolled in third grade in 1992-93 
completed the School Attitude Measure (American College Testing) (SAM) and 
the CTBS-U in reading, language, and math in each of three successive years. 
Pilot schools were matched with comparison schools. Significant positive 
changes in achievement occurred over the period of the study (1992/93 to 
1994/95) within the four efficacy schools and within the four non-efficacy 
schools selected for comparison. Math achievement rose more in the efficacy 
schools than in the comparison schools. Boys, African American students, and 
white students also experienced greater improvement in math achievement in 
efficacy schools than in comparison schools. Appendixes include: an 
assessment checklist; selection criteria for comparison schools; statistical 
analysis of paired (dependent) samples; charts comparing SAM of pilot 
schools, math scores and CTBS scores; and a comparison with the Detroit 
Study. (LH) 
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ABSTRACT 

The purpose of this paper is to share research related to the implementation and impact of 
the Efficacy process on teachers and students in four Sacramento City Unified School 
District pilot schools over a four year period. The study asked three research evaluation 
questions. (1) To what extent have teachers implemented the efficacy process, as 

measured by an Efficacy Level of Implementation rubric? (2) To what degree do students 
who take pan in the Efficacy process show growth in personal efficacy? (3) To what 
degree do schools which infuse Efficacy into their classrooms show improved performance 
on the Comprehensive Tests of Basic Skills (CTBS-U)? A level of Implementation Rubric 
was developed with the assistance of an Efficacy Institute national trainer. Students 
enrolled in third grade in 1992-93 completed the School Attitude Measure (American 
College Testing) and the CTBS-U in reading, language, and math in each of three successive 
years. Pilot schools were matched with comparison schools. Significant positive changes 
in achievement occurred over the period of the study (1992/93 to 1994/95) within the 
four Efficacy schools and within the four non-Efficacy schools selected for comparison. 
Math achievement rose more in the Efficacy schools than it did in the comparison schools. 
Boys, African American students, and white students also experienced greater 
improvement in math achievement in Efficacy schools than in the comparison schools. No 
statistically significant relationships were identified between SAM results and NCE 
scores, although correlations appeared to be weakly positive. Scattergrams did not appear 
to show any clear nonlinear relationships. 
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Introduction. The SCUSD was awarded a three year grant by the DeWitt Wallace 
Foundation (Reader’s Digest) to offer the Efficacy Institute Seminars to schools beginning 
in Fall 1992. The seminar, intended for teachers, focused on improving the academic 
performance of students through the development of positive attitudes toward learning. 
Four pilot schools received the initial five day seminar. The first four days involved 
presentation of theory and discussion of such topics as the innate ability and 
developmental models of learning as they operate at large and in schools. The final day 
organized teachers by grade level groups and focused on developing curriculum for 
students. By winter, the pilot schools had committed to serve as dissemination sources 
and to evaluate the effects of implementing efficacious processes in classrooms and 
throughout the school on students. In addition, the schools contracted for two additional 
visits (coaching and support) from Efficacy Institute Trainers. Trainers visited 
classrooms and met with teachers and administrators at each site. 

In the following two years, fifty schools voluntarily completed the Efficacy Seminar. The 
purpose of this paper is to share research related to the implementation and impact of the 
Efficacy process on teachers and students in the four elementary pilot schools over a three 
year period. 

Theoretical Perspective. Efficacy is defined as a judgment of one's personal 
capabilities (Bandura, 1986). It is viewed as a key element in student achievement of 
individual classroom tasks and mastery of subject matter in all disciplines. Self-efficacy 
finds its basis in the social-cognitive theory of Albert Bandura (1986). It is an 
individual's belief in his or her ability to perform a behavior in a given situation. People 
with a stronger sense of self-efficacy will decide to approach a complex situation due to 
their expectation of success. 

According to Bandura, self-efficacy beliefs are formed as a result of four . sources of 
etficacy information, the first being the most powerful and the last, the least powerful. 

Successful completion of a target behavior (inactive attainment). 

Observing or visualizing another person's completion of a target behavior, particularly 
when the model is perceived as being similar to him or her. 

• Verbal persuasion. 

• Physiological state 

Once information is conveyed, an individual weighs, selects, and integrates it into a self- 
efficacy judgment. 

An assumption often made in teaching is that if students are clear as to the outcome of a 
specific learning task and their behavior is clearly aligned with that outcome, there is an 
assurance ot task completion and probable successful execution of the task. An 

alternative explanation would be that if students hold a clear efficacy expectation for a 
task, they believe they can execute the behavior required to produce the outcomes. Como 



(1986) argues that students must develop self-efficacy first in order to use cognitive 
learning strategies for a task or in a particular subject area. Students must learn to 
manage effort in the face of particular difficulties. 

These difficulties can lie in the adult community (including racism and discrimination on 
the one hand and the failure to mobilize resources on the other). Obstacles can be 
external, external/institutional, intemal/psychological and/or cultural. Many times they 
are subtle. The Efficacy Institute takes the position that when honestly confronted and 
openly acknowledged, the obstacles can be handled. 



Teacher training models such as TESA (Teacher Expectations Increases Student 
Achievement) and Elizabeth Cohen's classroom collaboration model (1995) which 
emphasizes equal status interactions in classrooms both incorporate efficacious processes 
as an important variable in teacher behavior and. student achievement. 

Efficacy Institute Training and Implementation of Efficacious Processes. A 
well known and comprehensive curricular approach to student efficacy at the elementary, 
middle and high school levels is the work of Jeff Howard and the Efficacy Institute. This 
Massachusetts-based, non-profit organization provides seminars for school staff and other 
district personnel as well as ongoing coaching. Efficacy Institute seminars share 
theoretical constructs and research with teachers related to development and intelligence 
(Howard, 1991), attribution (Bernard, 1985; . Driscoll, 1986), goal setting and 
expectancy/performance models (Dwek), and power/motivation (McClelland, 1991). Based 
on this training and the belief that all students can develop their intelligence through 
application and hard work, teachers apply their learnings to their own school and 
classroom settings. 



The Efficacy Institute offers a classroom curriculum as a reference and a guide. 

Curriculum covers ten conceptual areas, which when applied are assumed to empower 
students with skills to manage and build confidence, take responsibility for and believe 
in their own learning processes. The teacher assists students in this development and 
avoids practices that imply that the student cannot "get smarter." Teaching staff also plan 
schoolwide activities which model the conceptual areas. 



Conceptual Areas 



Who Am I? 

Development 
Think You Can 
Get Smart 
The Learning Zone 

The Institute takes the position that while 
already have an awareness of and use a 



Using Feedback 
Obstacles and Problems 
Rise to a Higher Level 
think You Can. Get Smart 
Weak-Side. Strong-Side 

their curriculum is useful, most teachers 



variety of instructional strategies, curricular 
approaches and methods which can be called upon in building a comprehensive efficacious 
process in their classrooms. 



Research Questions. The pilot school research addressed one evaluation research 
question related to staff level of implementation of the Efficacy process and two evaluation 
research questions related to student outcomes. The study was longitudinal, following 

school staffs and students over a four year period beginning in the 1992-1993 school 
vear. 
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Staff Level of Implementation of the Efficacy Process: 

Question 1. Level of Implementation. To what extent have teachers implemented 
the efficacy process, as measured by an Efficacy Level of Implementation rubric? 

Student Outcomes: 

Question 2. Student Attitudes. To what degree do students who take part in the 
Efficacy program show growth in personal efficacy as measured by the School 
Attitude Measure (SAM)? 

Question 3. Student Achievement. To what degree do students in the four pilot 
schools show improved performance on the Comprehensive Tests of Basic Skills 
(CTBS-U)? 



Data Sources or Evidence 

Level of Implementation: 

Question L. Level of Implementation. To what extent have teachers implemented the 

efficacy process, as measured by an Efficacy Level of Implementation rubric? 

An implementation rubric (Appendix A) was developed by the District Accountability 
Office in concert with the national Efficacy lead trainer, Diane Jackson. 

The rubric includes five stages. At the "beginning”, stage of implementation, teachers can 
be observed using efficacious language in one subject area, students becoming familiar 
with the concepts, and begin to use the terminology. Teachers receive consultation and 
support for implementation from administrators and begin to discuss the ways in which 
students participate in class (including equity issues). Conversations with parents begin 
to include efficacious language and concepts. Teachers become more aware of the ways in 
which students are given feedback about their work. 

By the "growing development" level of implementation, Efficacous processes can be 
observed across all subject areas. Student learning is more self-regulated with students 

actively discussing their learning goals and work in relation to student achievement. 
Parents begin to discuss their students' progress in terms of these concepts. Teachers, 
students and parents utilize a variety of approaches to recognize student achievement and 
discuss academic progress. 

At the "full implementation” level. Efficacy is infused into all subject areas, and student 
learning is described as self-regulated, students actively discuss learning goals and work 
in relation to their achievement. Teachers utilize a variety of approaches to recognize 

student achievement. Parents are able to discuss their students’ progress in terms of 

these concepts. 

The rubric was field tested with the four pilot schools and subsequently validated by 
ongoing research in Kansas City. Kansas schools conducted by the Academy of Educational 
Development. New York. This group of researchers utilized ethnographic/observational 
methodology and analysis of student test scores to identify contextual features of schools 
which successfully apply the Efficacy process (1994). The rubric was administered to the 
Sacramento teachers each year in April. School principals also completed the rubric and 
participated in an in-depth interview. 



During each year of the project, the principal and each school staff met with the Project 

Evaluator to review and discuss the findings and implications of the yearly evaluation 
report. 

Student Outcomes: 

Question — 2*. Student Attitudes. To what degree do students who take part in the Efficacy 

process show growth in personal efficacy as measured by the School Attitude Measure 

(SAM)? 

The School Attitude Measure (American College Testing) was selected to answer this 

questions because of the close association of its scales with the Efficacy concepts as 

presented by the Efficacy Institute. 

School Attitude M ensure Subscales 

Motivation for Schooling. Students general feelings about the school experience, 

how hard students want to work in school, and how highly they value school. 

Academic Self-Concept-Performance Based. -Students feelings about their 
performance and confidence in their abilities. 

Academic Self-Concept-Reference Based. Students thoughts about how other people 
(teachers, family, friends) feel about their performance and abilities. 

Sense of Control Over Performance. Students feelings about their own 

responsibility for the outcomes of situations and events at school. 

Instructional Mastery. Students sense of their actual academic skills. 

All students at the grade level designated by the study took the survey in April of each 
school year. i.e. third graders in April of 1993, all fourth graders in April of 1994, and 
all fifth graders in April of 1995. Longitudinal summaries of individual student progress 
were prepared and distribute to teachers and principals n a yearly basis. 



Question — T Student Achievement. To what degree do students in the four pilot schools 

show improved performance on the Comprehensive Tests of Basic Skills (CTBS-U)? 

The CTBS-U scores of students with complete sets of Student Attitude and CTBS scores for 
the duration of the study were reviewed. 

Pilot Efficacy Schools: NCE scores of students who had complete sets of both SAM and 

CTBS scores for all three years of the program (1992/93 to 1994/95) were used. 

Matched NCE scores for each student were used to calculate three sets of changes in NCE 
scores: 2-year (1992 to 1994), first year (1992 to 1993) and second year (1993 to 1994). 
T-tests were performed on each set of paired differences to determine whether the average 
change in NCE was significant. 

Comparison Schools: A group of schools with socioeconomic status (SES) rankings and 

distribution of CTBS results similar to those of the four pilot Efficacy schools were 
identified. From this group, four schools which did not receive Efficacy training during 
1992-94 were selected to form a comparison group (see Appendix B). CTBS results of the 
comparison group who had a complete set of NCE scores for all three years were reviewed. 
Three sets of changes (2-year, first year, second year) in NCE scores were calculated and 
tested for statistical significance. 
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Pilot vs. Comparison Schools: T-tests were performed to determine whether pilot schools 

and comparison schools average improvements in NCE scores differed significantly from 
each other. Pilot and comparison school groups were not assumed to be dependent, i.e. 
their variances and means were not assumed to be equal (Appendix C). 



Results 



Staff Level of Implementation of the Efficacy Process: 

Question — L. Level of Implementation. To what extent have teachers implemented the 

efficacy process, as measured by an Efficacy Level of Implementation rubric? 

The Five Sections of the Rubric 
L Instruction 4. Assessment 

2. Resources & Materials 5. Teacher, Parent, School Relationship 

3. Classroom Organizational Strategies 6. Overall 



Principal Ratings of Staff 





Prior to 
Development 


fj^ 9 g inn i n g|j|? 
it© e ve lop men Bl 


Gfowing 5 ^^ 

^Development 


Aim os tr^Eu ||8f 
DevelopmeriliS 


|Zone|p 

8 Development 4 


1992-93 




X 








1993-94 






X 






1994-95 






X 







Both teachers and their principal completed the implementation rubric. Teacher ratings 
were shared with principals for discussion. Teacher ratings tended to be higher overall 
than principals. The results reported in the above table, reflects principal 
determinations of the level of implementation for staff at that site. In June of 1993, the 
four Principals reported that they were at the Beginning-Growing stages of 
implementation. By June 1994, the rating reflected "growing development" with a few 
teachers still in the "beginning development" stage. 

The principals felt that the Efficacy process gave teachers a common language for 
discussing issues,, encouraged risk-taking in making changes in their classrooms, and 
encouraged renewed focus on selecting/developing challenging meaningful learning tasks 
for students. They also shared that unless teachers personally live the process, it will not 
make a difference for their students. To be effective, it has to be tied to actual student 
achievement in each class. They observed that as teachers came to believe more strongly 
that their students were capable of doing well on specific tasks, their students began to do 
better at those tasks. For example, one principal noted that "some teachers now really 
believe that their students can become good writers and as a result, the students are 
writing better." 



Student Outcomes: 

Question 2_i_ Student Attitudes. To what degree do students who take part in the Efficacy 

process show growth in personal efficacy as measured by the School Attitude Measure 
(SAM)? 

Third grade students enrolled at the pilot schools in April 1993 completed the School 
Attitude Measure (SAM). The measure was given in April of the following year, 1994, to 
all fourth grade students, and again in April of 1995 to all fifth grade students. Sixty- 
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eight students had completed the SAM during all three years. The SAM was not 
administered to a comparison group. 



The graph displaying the means for these matched cases for the three years can be found 
in Appendix D. 

In April of 1993, the baseline means on four of the five subscales were above the national 
mean of 50 NCE. This result was discussed as follows in the subsequent evaluation report 
on Efficacy presented to the district. 

Local means may be higher than national means as a result of prior, programs in 
each school related to student attitudes and instructional mastery. 

Implementation of efficacy lessons and concepts from October through February 
could also be an influence. 

The question was posed. If SAM scores are already above the national mean, why 
do we need Efficacy? It was concluded that the Efficacy process builds teacher 
and. student capability culminating in improved academic achievement. It was 
expected that higher than average SAM scores would influence CTBS scores. 

At the first comparison point, April 1994, there were gains on four measures: 



The following discussion was included in the June 1994 evaluation report. 

Student growth in Instructional Mastery is considered important in terms of the 
Efficacy goal of incremental development. This subscale asked students to report 
on their actual school skills including ability to use time effectively, to focus 
attention and concentrate on tasks, to seek and use feedback, and to evaluate one’s 



Growth was across the board for both sexes. It is important to note that females 
consistently scored six to ten NCE points higher than males on all scales." 

The report concluded 

"The data show positive growth for third grade students who have been involved in 
the Efficacy process. This is encouraging given that this data reflects the first 
year of Efficacy implementation. Often, when new school-wide processes are 
adopted, effects of the processes do not become apparent until the second year of 
implementation." 

At the second comparison point, April 1995. the gains that did occur were minor. It was 
thought that the leveling of scores during the third year might be associated with the 
reduction in monetary and coaching support to these sites. The first year of 
implementation brought training, consultants and schoohvide implementation of the 
Efficacy processed; the second year, two Efficacy coach visits and a designated lead 
teacher at each site; the third year, no district support or grant funds. During this last 
period. Efficacy implementation was dealt with as each school could muster. In one 



Motivation for Schooling 
Self-Concept, Performance Based 
Self-Concept, Reference Based 
Sense of Control 
Instructional Mastery 



+4.2 NCE 
+2.0 NCE 
-0.8 NCE 
+2.0 NCE 
+6.9 NCE. 



own work. 



school, a lead teacher continued to direct implementation on her own time; in another, it 
was folded into a life skills emphasis, in another, the Principal continued working one- 
on-one with teachers. Two schools had large influxes of new staff members ( seven to nine 
teachers) in need of training. Receiving training was problematic due to lack of 
substitutes. New teachers had little familiarity with the concepts, terminology or 
classroom strategies. 



Student Outcomes: 

Question 3. Student Achievement. To what degree do students in the four pilot schools 
show improved performance on the Comprehensive Tests of Basic Skills (CTBS-U)? 

Significant positive changes in achievement occurred over the period of the study 
(1992/93 to 1994/95) within the four Efficacy schools and within the four non-Efficacy 
schools selected for comparison (Appendix E). Math achievement rose more in the 
Efficacy schools than it did in the comparison schools. Boys, African American students 
and white students also experienced greater improvement in math achievement in Efficacy 
schools than in the comparison schools. 

Pilot Efficacy Schools Results: Over the two year period from 1992 to 1994, student 
scores improved significantly in reading, language and math in the four pilot schools. The 
group demonstrated a 5.0 NCE point growth in reading, a 3.7 point growth in language and 
a 9.1 point growth in math (Appendix F). These improvements were significant at the 99%, 
95%. and 99% levels, respectively. 

During the first year, the average improvement in pilot Efficacy schools’ student scores 
was significant in all three areas. Students in these schools demonstrated a 3.9 point 
increase in reading, a 5.9 point increase in language and a 6.5 point increase in math. 

Each of these increases was statistically significant at the 99% level. 

During the second year of the Efficacy program in these schools, only math achievement 
showed significant growth. This increase. 2.6 NCE points, was significant at the 90% level. 

Comparison Schools Results: Over the 2-year period from 1992 to 1994, comparison 

school student scores improved significantly in ail 3 subject areas. Students in these 
schools demonstrated a 5.8 NCE point growth in reading, a 7.0 point growth in language 
and a 4.5 point growth in math (Appendix F). Each of these improvements was significant 
at the 99% level. 

During the first year, only language achievement showed significant growth in the non- 
Efficacy comparison schools group. Language NCE rose by 8.2 points, significant at the 
99% level. 

In the second year, comparison school reading and math achievement showed significant 
growth. Students demonstrated 3.1 NCE point growth in reading and 4.2 point growth in 
math, significant at 95% and 99%, respectively. 

Achievement Differences Between Pilot Efficacy Schools and Comparison Schools. Students 
in the four Efficacy schools demonstrated a greater improvement in math than did their 
counterparts in the four comparison schools during the two years of this study (1992/93 
to 1994/95). The average increase in Efficacy students’ math scores was 9.1 NCE points 
during this period, while comparison students' scores increased by only 4.5 points. The 
difference between the two, 4.6 NCE points, was significant at the 95% level. 
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Boys as a group showed a significantly greater improvement in math in the pilot Efficacy 
schools than they did in the comparison schools (Appendix G). Their math achievement 
rose by 11.4 NCE points over the 2-year period, while in the comparison schools, boys’ 
math achievement rose only by 4.2 points. The difference, 7.2 NCE points, was significant 
at the 95% level. 

Two ethnic groups had significantly greater growth in math NCE scores in the Efficacy 
schools: African American students and white students (Appendix H). African American 
students in the Efficacy schools experienced an average improvement of 8.4 NCE points, 

while African American students in comparison schools showed an increase of only 1.5 

points over the 2-year period. The difference between the two groups was significant at 
the 90% level. There were only 6 white students in the Efficacy school group. These 
students had an average increase in their NCE scores of 16 points, significantly greater (at 

95%) than the 2.8 unit increase of the 17 white students in the comparison schools. 

Comparison of SAM and CTBS Results : No statistically significant relationships were 

identified between SAM results and NCE scores, although correlations appeared to be 

weakly positive. Scattergrams did not appear to show any clear nonlinear relationships. 

No SAM pretest was given. The first SAM test was given to students in the pilot Efficacy 
schools in spring of 1993, after Efficacy was introduced and the CTBS-U was given in the 
fall of 1992. 

Ways of evaluating the relationship between SAM and CTBS results need to be further 
developed. Neither the Detroit study (1992) of the effects of the Efficacy process on 
students nor the present study found statistically significant relationships between the 
results of these two measures. This relationship is of interest because it is theorized that 
changes in student attitudes as a result of Efficacy programs precede improvements in 
achievement (Appendix I). 

Summary 

Did CTBS scores change over the course of the project for the pilot Efficacy schools? Yes, 
since in every subject there was a significant increase in achievement over the two years 
of the study. In all three subjects there was a significant increase in average student NCE 
scores during the first year. Only math NCEs rose significantly during the second year. 

What was the source of significant changes in NCE scores? Which classes within schools 
contributed to the change? This was looked at early in the analysis. Two bilingual 
classrooms evidenced the most dramatic growth on the attitude measure however, the 
analysis could not show that any one class was driving the results. Subdividing the sample 
into schools or classes resulted in very small sample sizes, greatly reducing the 
possibility of obtaining statistically significant results. 

Did the comparison schools CTBS scores change between 1992 and 1994? The scores did 
change. Average increases in comparison school NCE scores were statistically significant 
in reading, language and math over the two-year period, although only language 
achievement increased significantly in the first year and only reading and math increased 
significantly in the second year. 

Educational or Scientific Importance of the Study. Much of the research on self- 
etficacy measures small subsets of the construct (the constellation of cognitive schema's 
or networks of knowledge about the self) and. adds to our understanding of the possible 
relational schema’s students can develop for themselves related to achievement in subject 
matter areas. The research described here studied the broad construct of self-efficacy as 
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it occurred in the naturalistic setting of the school and classroom. The pilot schools 
focused on enhancing self-efficacy and academic achievement school wide over a period of 
years. This study and findings can be of assistance to district research and development 
departments in designing evaluation research which includes teacher and student efficacy 
as a construct. 

Implications and Extensions. The statistical analysis used for this study suggests 

ways in which useful information can be obtained quickly and clearly by testing paired 
samples. 

A benefit of the study is the development of the ‘Efficacy Level of Implementation’ rubric. 
This evaluation tool enhanced visualization, discussion, and measurement of 
implementation. In general, as more becomes known about implementing efficacious 
processes in classrooms, tools such as this can become more precise and specifically 
defined. 

How well students understand a program has a great effect on how successful the program 
is. An evaluation of students’ understanding of Efficacy language, concepts and goals 
might shed additional light on the extent to which students are reached by this program. 

Some Sacramento City USD schools now utilize highly defined and measurable subject 
specific programs such as Reading Recovery, Success For All. The structuring, clear 
articulation and assessment these programs incorporate also influences student sense of 
progress and self-efficacy. In addition, Sacramento City teachers now receive training in 

reading on a yearly basis. Future models should account for the effect of subject specific 
models of learning and teacher training on student attitudes and achievement in order to 
provide more information about the net effect of efficacious processes on student attitudes 
and achievement. Future models will utilize relational databases which can account for 
students who transfer schools or leave the district during the duration of the project. 
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EFFICACY LEVELS OF IMPLEMENTATION 
Assessment Checklist (Elementary School) 

Directions, nris checklist contains five sections related to implementation of Efficacy at your school: 

(1) Instruction ■■■•••• ■ .... - . : ... . - 

(2) Instructional Resources and Materials 

(3) Classroom Organizational Strategies - - 

(4) Assessment 

(5) Teacher, Parent, School Relationships. 

Please check all items which apply . (In some cases, you may find it appropriate to check more than one 
response per line.) Use this assessment checklist to foster discussions with your colleagues and to decide 
where you are beginning and how you want to proceed over the course of the semester/year. 



1. INSTRUCTION 



PRIOR TO 
DEVELOPMENT 


BEGINNING 


GROWING 


ALMOST FULL 


THE ZONE OF 


DEVELOPMENT 


DEVELOPMENT 


DEVELOPMENT 


OPTIMAL 

DEVELOPMENT 



□ Little or no visual 
evidence of Efficacy 
concepts. 



□ Some visual evidence 
of Efficacy concepts 
(fliers, banners, 
bulletin boards) in 
some classrooms. 



□ Visual evidence of 
Efficacy concepts in 
most classrooms. 



□ Visual evidence of 
Efficacy concepts 
throughout all 
classrooms. 



G Visual evidence of 
Efficacy concepts 
throughout ail 
classrooms. 



Teachers unsure/ 
unaware of Efficacy 
process. 


□ Teachers and staff 
using Efficacy 
language in one 
subject area. 


□ Teachers and staff 
really believe in 
Efficacy. Use 
language in one or 
more subject areas. 


□ Teachers and staff 
believe in Efficacy, 
and begin to infuse 
the language across 
all subject areas. 


C Teachers believe in 
Efficacy and the 
language is infused 
across all subject 
matter areas. 


Students unaware of 
Efficacy process. 


□ Students are familiar 
with terminology but 
may not have it 
internalized. 


□ Students are 
beginning to 
internalize and use 
the Efficacy 
concepts. 


G Students actively use 
Efficacy language 
and concepts. 


G Students and 

teachers actively use 
Efficacy language 
and concepts. 


Teachers are teaching 
as best they know 
how. They need 
insevice. 


□ Teachers actively 
internalizing and 
learning Efficacy. 
Need consultation 
and support. 


D Focus of discussion 
is on application. 

"How can I continue 
to enhance the process 
of implementing the 
Efficacy curriculum?” 


G Focus of discussion 
is on how to generate 
consistent 
improvement in 
students by 
structuring 
incrementally more 
challenging lessons. 


G Teachers generate 
rapid development in 
their classrooms 
through curricular 
strategies. 


Teachers and 
students threatened 
by feedback to make 
changes. 


□ Teachers and 
students 
acknowledge 
feedback as a 
developmental 
process. 


□ Teachers- and 

students actively use 
feedback as a 
developmental 
process. 


G Administrator, 

teachers and students 
actively use feedback 
as a developmental 
process. 


G Administrators, 

teachers, students and 
parents actively use 
feedback as a 
developmental 
process. 
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EL INSTRUCTIONAL RESOURCES AND MATERIALS 



t'KIOR TO 

development 


BEGINNING 

DEVELOPMENT 


GROWING 

DEVELOPMENT 


ALMOST FULL 
DEVELOPMENT 


THE ZONE OF 

OPTIMAL 

DEVELOPMENT 


O Teachers not using 
Efficacy curriculum 
manual and 
guidelines. 


□ Teachers actively 
discuss and question 
Efficacy curriculum. 


□ Teachers have less 
dependency on 
Efficacy curriculum 


□ Teachers writing, - 
implementing and 
evaluating their own 


□ Teacher developed 
Efficacious learning 
_ ^strategies are infused ‘ 




mqtfriafg Q 3 enhance 
instruction. Some - -1- 
development of own ~ 
Efficacy strategies - 


Efficacy canicular 
strategies^-;,-- - ■ 


.. across the gnrrindmn 



m. CLASSROOM ORGANIZATIONAL STRATEGIES 



PRIOR TO 
DEVELOPMENT 


BEGINNING 

DEVELOPMENT 


GROWING 

DEVELOPMENT 


ALMOST FULL 
DEVELOPMENT 


THE ZONE OF 

OPTIMAL 

DEVELOPMENT 


□ Teachers use 

disciplinary measures 
to maintain quiet, 
orderly classrooms. 
Teachers refer to 
student character or 
past transgressions 


□ Teachers begin 
integrating Efficacy 
into classroom 
organizational 
strategies. 


□ Relaxed friendly, 
harmonious 
classroom. Rules 
based on 
development. 


□ Classroom control 
based on 
development. 
Students take 
development 
seriously. Low 
incidence of 
misbehavior. 


□ Students monitor and 
assess their own 
development. 


Cj S tudents may or mav 
not be working in 
groups. Groups 
reflect innate ability 
model. Teachers 
need inservice. 


□ Students may or may 
not be working in 
groups. Teachers 
need support 
implementing 
developmental 
grouping strategies. 


□ Teachers researching 
classroom 
organizational 
strategies, Le., 
cooperative learning 
and heterogeneous 
groupings. Teachers 
field testing these 
strategies. 


□ Teachers are 

thoroughly familiar 
with a variety of 
classroom 
organizational 
strategies. Teachers 
utilize these as 
needed and 
appropriate. 


□ Full collaboration 
among teachers about 
classroom 
organizational 
strategies. 


□ Discrepancies in the 
participation of 
different groups of 
students in the 
classroom, i.e., 
bovs/girls, different 
ethnic groups, 
different ability/skill 
levels. 


□ Teachers become 
aware of 

discrepancies in the 
participation of 
different groups of 
students in the 
classroom, i.e., 
boys/giris, different 
ethnic groups. 


□ Student participation 
in ciasssroom. 
Evidence of active 
learning, engagement 
of ail students, 
reduction of 
discrepancies 
between ethnic 
groups. 


D Student participation 
in ciasssroom. 

Active learning, . 
engagement of all 
students, reduction of 
discrepancies 
between ethnic 
groups. 


□ Full student 
participation in 
classrooms. 



different ability/skill 
levels. 




best copy available 
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IV. ASSESSMENT 



PRIOR TO 
DEVELOPMENT 


BEGINNING 


■- GROWING 


ALMOST FULL 


THE ZONE OF 


DEVELOPMENT 


DEVELOPMENT 


DEVELOPMENT 


OPTIMAL 

DEVELOPMENT 





□ Teachers locked into 
the traditional "A-F* 
grading system. 
Incomplete records or 
student progress may 
occur. 



Teachers struggling --- Q 
with the grading/..; 
system, making some 
adjustments, Le-T^TT-” 
portfolios in some — --7 
areas, traditional - 
grading,such as . ~~ 

chapter tests, in -' : - 

others. 



Teachers - 

implementing 
altemativ ft grading 

strategies,^ tests* 
portfolios, in some — 
subject areas. 



□ Teachers 
implementing 
alternative grading 
strategies, using a 

variety of methods, 

Le„ separate grades 

for structure and 

content, checks and 
minuses, highlighting 
what needs to be 
worked on, 
portfolios, etc. 



The school institu tes 
an alternative grading 
system* 



Teachers need site 
support with 
alternative 
assessment systems. 



Teachers are waiting 
for site support of 
their efforts. 



□ Teachers are 
receiving some site 
suport for alternative 
grading strategies. 



Teachers are 
receiving full site 
support with 
alternative grading 
strategies. 



An alternative 
grading system 
receives 

administrative and 
parent support. 



Teachers need site 
support with giving 
feedback and 
recognition to student 
progress in academic 
and nonacademic 
areas. 



□ Teachers struggling 
with giving feedback 
and recognition to 
student progress in 
academic and 
nonacademic areas. 



Teachers giving some .□ 
feedback and 
recognition to student 
progress in academic 
and nonacademic 
areas. 



Teachers giving 
ongoing feedback 
and recognition to 
student progress in 
academic and 
nonacademic areas. 



Teachers and other 
school staff giving 
ongoing feedback 
and recognition to 
student progress in 
academic and 
nonacademic areas. 



V. TEACHER, PARENT, SCHOOL RELATIONSHIPS 



PRIOR TO 
DEVELOPMENT 


BEGINNING 


GROWING 


ALMOST FULL 


THE ZONE OF 


DEVELOPMENT 


DEVELOPMENT 


DEVELOPMENT 


OPTIMAL 

DEVELOPMENT 



□ Teacher-parent 
discussions reflect 
the innate ability 
model. 



□ Teacher-parent 
discussions begin to 
reflect the 
developmental 
modeL 



□ Teachers frame 
discussions with 
parents around 
Efficacy concepts. 



□ Teachers and parents 
use Efficacy concepts. 
They freely discuss 
student development. 



□ Teachers, parents, 
and school staff have 
a common 
Efficacious base for 
discussing student 
development and 
achievement. 



Communication with 
parents is sporadic 
and focused on 
behavior. 



□ Teachers have a 
system in place to 
inform parents of 
student effort and 
how it relates to 
achievement. 



E M, BEST COPY AVAILABLE 



Communication with 
parents is more 
consistent. 
Administrators and 
staff identify a model 
for communicating 
with a wide audience 
of parents. 



Teachers offer get- 
togethers for parents 
about Efficacy. 



□ Teachers, parents and 
administrators offer 
regularly scheduled 
Efficacious activities. 
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VL PULLING IT ALL TOGETHER 
WHERE ARE YOU OVERALL? 



□ PRIOR TO 

DEVELOPMENT 





□ BEGINNING □ GROWING 
DEVELOPMENT DEVELOPMENT 




□ ALMOST FULL 
DEVELOPMENT 



□ THE ZONE OF 
OPTIMAL 
DEVELOPMENT 



THANK YOU VERY MUCH FOR COMPLETING THIS CHECKLIST ! 



Comments: 



RH/de— Sac, CA 
12-93 
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Appendix B 



Selection of Comparison Schools 

Comparison schools were not selected at the beginning of the project, because school 
participation in the Efficacy program was entirely voluntary. Which schools would be 
adopting Efficacy and in what year was not known in 1992. Forty-nine schools received 
some Efficacy training after 1991/92 and 30 of them eventually adopted Efficacy. 

Socioeconomic status, or SES ranking was chosen as a basis for identifying a group of non- 
Efficacy schools with similar poverty status as the four pilot schools. SES ranking is a 
criterion that is used in setting district achievement targets for the Sacramento City 
Unified School District. 

The correlation between SES ranking and reading achievement (in NCE points) is 
historically high. In 1993/4, the correlation between SES rank and reading achievement 
was 89%, and between SES rank and math achievement the correlation was 84%. 

Three indicators are used to construct a school’s SES index. First, each school in the 
district is given a ranking based on the proportion of students in that school who come 
from families receiving AFDC. Next, each school is assigned a rank based on the 
proportion of its students receiving Free/Reduced Lunch program services. Finally, each 
school is assigned a rank based on its stability index, or the proportion of students who 
were in continuos enrollment from October through June. These three indices are 
averaged, and the result is used to rank schools in “SES order.” Each school’s achievement 
targets for each school are set based on the average achievement of the 5 schools above and 
the 5 below it in SES rank. 

To select comparison (non-Efficacy) schools, pilot school SES rank and SES components' 
were compared with non-Efficacy schools’ SES rank and components. This narrowed down 
possible candidates for a comparison group to 6 schools. 

Reading, language and math NCE scores from the CTBS have been presented by Sacramento 
City Unified School District in recent years in a “four quarters of the distribution” graph. 
This four quarters distribution displays the percent of NCE scores for each subject and 
grade that fall between the Oth, 25th, 50th, 75th and 100th percentiles. In 1992 four 
quarters data was available for each school in the district. This “four quarters” data has 
been used in the past by the district to identify schools with similar needs for support. 

The final 4 schools selected for the comparison group were chosen based on how well their 
four quarters distribution matched that of the pilot schools, both as a group and 
individually. 

Schools that implemented or received training in Efficacy during 1992/93 through 
1994/95 were excluded from consideration. Some schools that would have made good 
comparison schools, based on SES rank and distribution of CTBS scores, implemented 
Etficacy during the period of this study and could not be used for the comparison group. 



Thank you to Paula A. Mara, M.S. for assistance with comparison sample selection and for 
the methodology and statistical analysis of sample and comparison groups. 
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Appendix C 



Statistical Analysis of Paired (Dependent) Samples: testing for significant 

change when you have multiple observations over time from the same subject) 

Example, pre-test and post-test after a special program has been implemented. 

When you can use this approach? 

The two populations you are drawing from have different means but each population has 
the same variance. Thus correlation may exist between the means of samples drawn from 
the two populations. 

Example, sample 1 is the test scores of a group of students before receiving special 
instruction, sample 2 is the test scores for the same students after the instruction. 

Data is in the form of matched pairs of one student's scores “before and after.” 

The hypothesis to be tested is whether the means of the two populations are equal or not: 



Hq: m^ - m 2 = 0 
Hq: mj - m 2 K 0 



Method: compute the test statistic 

t = (mean difference)/{(std.dev. of difference)/(square root of sample size)}. 
Compare this calculated value with values from the t distribution for the appropriate 
significance level (l-alpha/2) and degrees of freedom (n-1). 

Computation steps: 

1. Find differences d and mean difference dbar. Lower case d is the difference “before and 
after” and mean difference dbar is the mean of those differences. 

2. Find the standard deviation of the difference s d = -r S(d - Sd / n)2/(n - 1) 

3. Construct t = (Sd / n)/(sd / + n). Compare it to the t-table in order to determine 
significance level for appropriate degrees of freedom. 



XI (before) 


X2 (after) 


d = XI - X2 


(d - dbar)2 


1 0 


1 5 


5 


1 6 


1 2 


1 3 


1 


0 


6 


6 


0 


1 


1 2 


1 0 


. 2 


9 



n = 4 sum 4 2 6 



mean dbar 1 

The sum of the squared differences is 26 and n-1 is 3, so the standard deviation sd is the 
square root of (26/3), or 2.9439. The resulting t-statistic is 0.6794, which is not 
significant with n-1 = 3 degrees of freedom. We would need a calculated t-statistic of 3.18 
or greater to have significance at the 95% level with only 4 observations. 



The NCE is distributed approximately as a uniform distribution. The Central Limit 
Theorem assures us that when a sample size is large, the sample means are distributed 
approximately normally. This is the rule that allows us to use the t-statistic. 
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Pilot Schools, Three-Year SAM Comparison (n=68) 
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2-Year Improvements in Math NCE Scores Were Significantly Greater for Pilot Schools Than for 
Comparison Schools - As a Group, for Boys, for Black Students, for White Students 
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Appendix I 

Comparison of the Detroit Study with the Sacramento Study 

achievement S ™ * l' 5 "” S '" dy thaI doc “ ,I,eriIcd changes in CAT (California 
scores were reported To f ” CXpenmcmal and comro > groups. Pre- and post-Efficacy 
S a “ticlllv Zi 1 J, 8 'T and s,a,is,lcally significant changes were identified. 

ideZ ed Un fs of 1, enC ' S T' ,W “ n CXperim '"‘ al a " d control groups were also 
identified. Units of measurement included scale and GEUs (grade equivalent units), 

comparison s“htol 7^' P«. Tn^Zt'efficaT'" 1 '' 1 ‘h” 8 " ^ CTBS f ° r Pi ' 0t 

statistically significant differences between pTo and comnanTon' reP ° r1 " 1 ** T" gr °“ P; 
identified. Units measurement were NCE percemile scZes 8r °" PS ^ 

Detroit findings: 

Reading: Both groups showed increase in GEUs over a 1-vear npHnH h > 

students increased more than controls. ' P ’ * ex Penmental 

control b s 0th ST ° UPS ^ ’ nCreaSed GEUs ' experimental .students’ GEUs increased more than 
Sacramento findings: 

sZnS„. B Zcre 8 a” UP i?vlrT eiS bT NC& ° Ver ,he 2 ' ycar p ™ d ' »">• schools showed a 

year 2 There was no sivn f 7 d n _ corn P anson schools showed a significant increase in 

year 2. There was no significant difference results between the two groups of schools. 

“gtkrZZcZtLTsr 1 '”' inc " ase ° VCr ,he 2 ' yca ' p ' ri °« «'<» schools had a 
second. In Math there was sicniffcmtl' 0 ”^ 50 " SCh °° ls had a significant increase in the 
than there was^e P " dt 

significant LtlT ^ ™ ** ~ ^ ^ “ a 

Conclusions : 

wZidT'm T'“ ar ' C ° nSiS ‘ em wi,h lhe findings of the Detroit study. Neither study 
studem 6 achievement. Sta,,S, Ca y S,S " ifiCa,,t "' aIi ‘"' S >" P - SaA and tneasLJ of 
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